feat: Add Zvec in-process vector database for semantic code search#5245
feat: Add Zvec in-process vector database for semantic code search#5245oldschoola wants to merge 5 commits into
Conversation
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
roboomp
left a comment
There was a problem hiding this comment.
review:p2 — coherent feature, but it is a new default-enabled native-backed code-search path and needs maintainer sign-off.
Blocking findings: duplicate file contents collide on Zvec document IDs, and stale chunks remain when an indexed file later becomes too large/binary. Also flagged unreliable post-topK pattern filtering and a full-suite-safety issue in the env override test.
Verification: bun test test/zvec-store.test.ts, bun test src/tools/__tests__/code-search-index.test.ts, and both package bun run check:types commands pass; a focused duplicate-ID repro shows Zvec stores only one doc for two same-ID chunks.
Thanks for the substantial implementation and the targeted smoke coverage.
…search Add native support for Zvec (@zvec/zvec), Alibaba's in-process vector database, as a code search backend. No MCP server or external daemon required — runs entirely in-process like the existing SQLite memory engine. mnemopi package: - ZVecCodeStore: vector store wrapper with HNSW vector search, BM25 full-text search, and hybrid (RRF) search for code chunks - zvec-config: environment-configurable settings (OMP_ZVEC_* env vars) - @zvec/zvec added as optional peer dependency coding-agent package: - code_search tool: workspace indexing with gitignore-aware scanning, 100-line chunking with 10-line overlap, embedding via mnemopi's embed()/embedQuery(), hybrid vector+FTS search with FTS-only fallback - Discoverable tool (loadMode=discoverable), only loads when needed - Incremental indexing: compares persisted chunk hashes to skip unchanged files across sessions; removeFile uses deleteByFilterSync for O(1) cleanup Verified with real @zvec/zvec native addon under Bun: create, upsert, vector search, FTS search, hybrid multiQuery, removeFile, close. 14 mnemopi tests + 2 coding-agent index tests pass, 0 fail.
- Add tools.codeSearchEnabled setting (default: false) to settings schema under Tools → Available Tools; code_search tool is now gated by this setting instead of the OMP_ZVEC_ENABLED env var - Add /code-index slash command to explicitly build or rebuild the Zvec code search index; uses loadMnemopi() for lazy module loading - Refactor indexing logic into standalone indexWorkspace() function in code-search-index.ts, shared by both the slash command and the tool - Tool no longer auto-indexes on first search; only searches the existing index and tells users to run /code-index when the index is empty - Update tool description to mention /code-index requirement
- Fix doc-ID collision: include file path in chunk hash (Bun.hash(filePath + "\0" + content)) so duplicate files don't overwrite each other in the Zvec index - Fix stale chunks: move currentFiles.add() after size/binary checks; call store.removeFile() when a previously-indexed file becomes too large or binary - Fix pattern filter: overfetch (topK*5, min 100) when a glob pattern is provided, then slice to topK after filtering - Fix test env safety: pass local env object to config helpers instead of mutating process.env
- Sort imports and fix formatting in all new files (Biome) - Remove unused private session field in CodeSearchTool - Rename unused signal parameter to _signal in #search()
460ed58 to
e750bc3
Compare
Zvec upsert/optimize/searchFts calls are synchronous but slow on CI hardware. Both tests exceeded the 5s default timeout (6.7s and 7.1s). Set describe-level testTimeout to 15s.
e750bc3 to
ff0d865
Compare
Summary
Add native support for Zvec (Alibaba's in-process vector database) as a semantic code search backend. No MCP server or external daemon — runs entirely in-process, just like the existing SQLite-based mnemopi memory engine.
Opt-in by design: Code search is disabled by default. Users enable it via
/settings→ Tools → Available Tools → Code Search (Zvec), then explicitly build the index with/code-index. Thecode_searchtool does not auto-index — it only searches the existing index and tells users to run/code-indexwhen the index is empty.Architecture
Changes
mnemopi package (
@oh-my-pi/pi-mnemopi)zvec-store.ts:ZVecCodeStoreclass — Zvec vector store wrapper with upsert, vector search (HNSW), FTS search (BM25), hybrid search (RRF),removeFile(viadeleteByFilterSync),getFileHashes, optimizezvec-config.ts: Config functions (OMP_ZVEC_*env vars for index path, chunk size, overlap, top-K)@zvec/zvecadded as optional peer dependencycoding-agent package (
@oh-my-pi/pi-coding-agent)settings-schema.ts: Addedtools.codeSearchEnabledsetting (default:false) under Tools → Available Toolsbuiltin-registry.ts: Added/code-indexslash command — explicitly builds or rebuilds the Zvec code search index; usesloadMnemopi()for lazy module loadingcode-search-index.ts: StandaloneindexWorkspace()function — shared by both the slash command and the tool; usesloadMnemopi()to keep mnemopi off the startup module graphcode-search-helpers.ts: Shared helpers (chunking, language detection, binary detection, file filtering)code-search.ts:CodeSearchTool— searches the existing index only (no auto-indexing); hybrid vector+FTS search with FTS-only fallback; overfetches when a glob pattern filter is providedcode-search.md: Tool description prompt (mentions/code-indexrequirement)loadMode=discoverable), only loads when neededcreateIfreturns null whentools.codeSearchEnabledis false or@zvec/zvecnative addon not loadableInfrastructure
@zvec/zvec: 0.5.0added to workspace catalogtrustedDependencies: ["@zvec/zvec"]added to root package.json (Bun lifecycle script allowlist)Key design decisions
tools.codeSearchEnableddefaults tofalse. Users must explicitly enable it in settings and run/code-indexto build the index. The tool does not auto-index on first search.embed()/embedQuery()pipeline (local fastembed via ONNX or API-based). Does NOT use pi-ai (no embedding client) or Zvec's Python-only embedding functions.getFileHashes()to skip unchanged files across sessions.removeFileusesdeleteByFilterSyncfor O(1) cleanup. Files that grow past 1MB or become binary have their stale chunks removed automatically.${Bun.hash(filePath + "\\0" + content)}-${lineStart}— includes file path to prevent duplicate-content collisions; Zvec rejects:,/,|in document IDs./code-indexslash command andindexWorkspace()useloadMnemopi()fromsrc/mnemopi/state.ts(memoized lazy loader) instead of rawawait import(), respecting onnxruntime-node NAPI load-ordering constraints.Verification
ZVecCreateAndOpen→ insert → optimize → vector query → FTS query → hybrid multiQuery → removeFile → close — all pass under Bunbun run check:typesReview fixes addressed
Bun.hash(filePath + "\\0" + content)) so duplicate files don't overwrite each othercurrentFiles.add()after size/binary checks; callstore.removeFile()when a previously-indexed file becomes too large or binarytopK*5, min 100) when a glob pattern is provided, then slice totopKafter filteringprocess.envTest plan
cd packages/mnemopi && bun test test/zvec-store.test.ts— 14 testscd packages/coding-agent && bun test src/tools/__tests__/code-search-index.test.ts— 2 testscd packages/mnemopi && bun run check:typescd packages/coding-agent && bun run check:types